Apparatus and method for generating an overview image of a plurality of images using a reference plane

ABSTRACT

An apparatus for generating an overview image of a plurality of images comprises a storage unit and an image processor. The storage unit stores a plurality of processed images of the overview image and is able to provide the overview image containing the plurality of processed images at their assigned positions for displaying. The image processor determines feature points of a new image and compares the determined feature points of the new image with feature points of a stored processed image to identify common feature points and to obtain 3-dimensional positions of the common feature points. Further, the image processor determines common feature points located within a predefined maximum distance of relevance to a reference plane based on the 3-dimensional positions of the common feature points to identify relevant common feature points. Further, the image processor processes the new image by assigning the new image to a position in the overview image based on a comparison of an image information of each relevant common feature point of the new image with an image information of each corresponding relevant common feature point of the stored processed image without considering common feature points located beyond the predefined maximum distance of relevance to the reference plane.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from European Patent Application No.10174019.9, which was filed on Aug. 25, 2010, and is incorporated hereinin its entirety by reference.

BACKGROUND OF THE INVENTION

Embodiments according to the invention relating to the field of imageprocessing and particularly an apparatus and a method for generating anoverview image of the plurality of images.

Much research has been done in the area of mosaicking of aerial imageryand surveillance over the past years. Many approaches have been proposedranging from using low altitude imagery of stationary cameras and UAVs(unmanned aerial vehicle) to higher altitudes imagery captured fromballoons, airplanes, and satellites. High altitude imagery and on-groundmosaicking such as panoramic image construction are dealing withdifferent challenges than low altitude imagery.

There has been a breakthrough regarding the seamless stitching in pastyears by exploiting robust feature extraction methods (see for example“Y. Zhan-long and G. Bao-long. Image registration using rotationnormalized feature points. In ISDA '08: Proceedings of the 2008 EighthInternational Conference on Intelligent Systems Design and Applications,pages 237-241, Washington, D.C., USA, 2008. IEEE Computer Society.”, “D.Steedly, C. Pal, and R. Szeliski. Efficiently registering video intopanoramic mosaicks. In Proceedings of the Tenth IEEE InternationalConference on Computer Vision, volume 2, pages 1300-1307, Los Alamitos,Calif., USA, 17-21 2005. IEEE Computer Society.”, “H. Bay, A. Ess, T.Tuytelaars, and L. Van Gool. Speeded-Up Robust Features (SURF). Comput.Vis. Image Underst., 110(3):346-359, 2008”), depth-maps (see for example“S. B. Kang and R. Szeliski. Extracting view-dependent depth maps from acollection of images. Int. J. Comput. Vision, 58(2):139-163, 2004”, “C.Cigla and A. A. Alatan. Multi-view dense depth map estimation. InIMMERSCOM '09: Proceedings of the 2nd International Conference onImmersive Telecommunications, pages 1-6, ICST, Brussels, Belgium, 2009.ICST Institute for Computer Sciences, Social-Informatics andTelecommunications Engineering”), 3D reconstruction of the scene, imagefusion, and many other approaches (e.g. “R. Szeliski. Image alignmentand stitching: a tutorial. Found. Trends. Comput. Graph. Vis.,2(1):1-104, 2006”, “H.-Y. Shum and R. Szeliski. Construction andrefinement of panoramic mosaicks with global and local alignment. InProceedings of Sixth International Conference on Computer Vision, pages953-956, 1998.”). A SURF feature-based algorithm is for exampledescribed in “H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. Speeded-UpRobust Features (SURF). Comput. Vis. Image Underst., 110(3):346-359,2008”. Results look seamless at the stitching part but a drawback isthat the transformation performed on the images leads to a distortion inscales and relative distances. Lines which are parallel in reality, arenot parallel anymore in the stitched image. This type of erroraccumulates over multiple images if not compensated. Such a traditionalfeature-based approach is difficult for some applications. For example,the generation of a geo-referenced image is hardly possible due to thescale and angle distortions as well as the error propagation overmultiple images. Other stiching algorithms are shown in “H.-Y. Shum andR. Szeliski. Construction and refinement of panoramic mosaicks withglobal and local alignment. In Proceedings of Sixth InternationalConference on Computer Vision, pages 953-956, 1998”, “Y. Furukawa and J.Ponce. Accurate Camera Calibration from Multi-View Stereo and BundleAdjustment. In Proceedings of IEEE Conference on Computer Vision andPattern Recognition, number 3, pages 1-8, Hingham, Mass., USA, 2008” or“G. Sibley, C. Mei, I. Reid, and P. Newman. Adaptive relative bundleadjustment. In Robotics Science and Systems (RSS), Seattle, USA, June2009”.

A challenge of low altitude imagery and mosaicking for surveillancepurposes is finding an appropriate balance between seamless stitchingand geo-referencing under consideration of processing time and otherresources. The scale difference as a result of different flying altituderesulted in several stitching errors. In other words, significantstitching errors induced by scale differences among images may bevisible. Similar objects may have different sizes, and there may be adisparity in horizontal and vertical stitching. A similar error mayoccur by inaccurate camera position or rotation. In other words,stitching disparities may be caused by inaccurate camera angle orposition.

Many approaches have been proposed to tackle these problems. Examplesinclude the wavelet-based stitching “C. Yuanhang, H. Xiaowei, and X.Dingyu. A mosaick approach for remote sensing images based on wavelettransform. In WiCOM '08: Proceedings of the Fourth InternationalConference on Wireless Communications, Networking and Mobile Computing,pages 1-4, 2008”, image registering in binary domains “X. Han, H. Zhao,L. Yan, and S. Yang. An approach of fast mosaick for serial remotesensing images from UAV. In FSKD '07: Proceedings of the FourthInternational Conference on Fuzzy Systems and Knowledge Discovery, pages11-15, Washington, D.C., USA, 2007. IEEE Computer Society”, automaticmosaicking by 3D-reconstruction and epipolar geometry “L. Lou, F.-M.Zhang, C. Xu, F. Li, and M.-G. Xue. Automatic registration of aerialimage series using geometric invariance. In Proceedings of IEEEInternational Conference on Automation and Logistics, pages 1198-1203,2008”, exploiting known ground reference points for distortioncorrection “P. Pesti, J. Elson, J. Howell, D. Steedly, and M.Uyttendaele. Low-cost orthographic imagery. In GIS '08: Proceedings ofthe 16th ACM SIGSPATIAL international conference on Advances ingeographic information systems, pages 1-8, New York, N.Y., USA, 2008.ACM”, IMU-based multi-spectral image correction “A. Jensen, M. Baumann,and Y. Chen. Low-cost multispectral aerial imaging using autonomousrunway-free small flying wing vehicles. Geoscience and Remote SensingSymposium, IGARSS, 5:506-509, 2008”, combining GPS, IMU and videosensors for distortion correction and geo-referencing “A. Brown, C.Gilbert, H. Holland, and Y. Lu. Near Real-Time Dissemination ofGeo-Referenced Imagery by an Enterprise Server. In Proceedings of 2006GeoTec Event, Ottawa, Ontario, Canada, June 2006” and perspectivecorrection by projective transformation “W. H. WANG Yue, WU Yun-dong.Free image registration and mosaicking based on tin and improvedszeliski algorithm. In Proceedings of ISPRS Congress, volume XXXVII,Beijing, 2008”. Some of these approaches are considering higher altitude“A. Brown, C. Gilbert, H. Holland, and Y. Lu. Near Real-TimeDissemination of Geo-Referenced Imagery by an Enterprise Server. InProceedings of 2006 GeoTec Event, Ottawa, Ontario, Canada, June 2006”,“X. Han, H. Zhao, L. Yan, and S. Yang. An approach of fast mosaick forserial remote sensing images from UAV. In FSKD '07: Proceedings of theFourth International Conference on Fuzzy Systems and KnowledgeDiscovery, pages 11-15, Washington, D.C., USA, 2007. IEEE ComputerSociety.”, “L. Lou, F.-M. Zhang, C. Xu, F. Li, and M.-G. Xue. Automaticregistration of aerial image series using geometric invariance. InProceedings of IEEE International Conference on Automation andLogistics, pages 1198-1203, 2008”, P. Pesti, J. Elson, J. Howell, D.Steedly, and M. Uyttendaele. Low-cost orthographic imagery. In GIS '08:Proceedings of the 16th ACM SIGSPATIAL international conference onAdvances in geographic information systems, pages 1-8, New York, N.Y.,USA, 2008. ACM”, “W. H. WANG Yue, WU Yun-dong. Free image registrationand mosaicking based on tin and improved szeliski algorithm. InProceedings of ISPRS Congress, volume XXXVII, Beijing, 2008”, whileothers are using different types of UAVs such as small fixed wingaircrafts “G. B. Ladd, A. Nagchaudhuri, M. Mitra, T. J. Earl, and G. L.Bland. Rectification, geo-referencing, and mosaicking of images acquiredwith remotely operated aerial platforms. In Proceedings of ASPRS 2006Annual Conference, page 10 pp., Reno, Nev., USA, May 2006”, “A. Jensen,M. Baumann, and Y. Chen. Low-cost multispectral aerial imaging usingautonomous runway-free small flying wing vehicles. Geoscience and RemoteSensing Symposium, IGARSS, 5:506-509, 2008.”, “Y. Huang, J. Li, and N.Fan. Image Mosaicking for UAV Application. In KAM '08: Proceedings ofthe 2008 International Symposium on Knowledge Acquisition and Modeling,pages 663-667, Washington, D.C., USA, 2008. IEEE Computer Society”.These aircrafts show less geo-referencing accuracy caused by higherspeed and degree of tilting (higher amount of roll and pitch). “Z. Zhu,E. M. Riseman, A. R. Hanson, and H. J. Schultz. An efficient method forgeo-referenced video mosaicking for environmental monitoring. Mach. Vis.Appl., 16(4):203-216, 2005” performed an aerial imagery mosaickingwithout any 3D reconstruction or complex global registration. Thatapproach uses the video stream which was taken from an airplane. “Y.Huang, J. Li, and N. Fan. Image Mosaicking for UAV Application. In KAM'08: Proceedings of the 2008 International Symposium on KnowledgeAcquisition and Modeling, pages 663-667, Washington, D.C., USA, 2008.IEEE Computer Society” performed a seamless feature-based mosaickingusing a small fixed-wing UAV. “J. Roβmann and M. Rast. High-detail localaerial imaging using autonomous drones. In Proceedings of 12th AGILEInternational Conference on Geographic Information Science: Advances inGIScience, Hannover, Germany, June 2009”. also used small-scalequadrocopters. The mosaicking results are seamless but lackinggeo-referencing.

“Howard Schultz, Allen R. Hanson, Edward M. Riseman, Frank Stolle,Zhigang Zhu, Christopher D. Hayward, Dana Slaymaker. A System forReal-time Generation of Geo-referenced Terrain Models. SPIE EnablingTechnologies for Law Enforcement Boston, Mass., Nov. 5-8, 2000” and“Zhigang Zhu, Edward M. Riseman, Allen R. Hanson, Howard Schultz. Anefficient method for geo-referenced video mosaicking for environmentalmonitoring. Machine Vision and Applications (2005) 16(4): 203-216”describes a system for generating 3D structures from aerial imagesuseing laser sensor to precisely measure the elevations. For this, alarger airplane with two camera systems is needed.

In “M. Brown and D. G. Lowe. Recognising Panoramas. In Proc. ICCV 2003”a purely image-based mosaicking is shown. It describes an automaticapproach for feature (using SIFT) and image matching by assuming thatthe camera rotates about its optical center.

“WU Yundong, ZHANG Qiangb, LIU Shaoqind. A CONTRAST AMONG EXPERIMENTS INTHREE LOW-ALTITUDE UNMANNED AERIAL VEHICLES PHOTOGRAPHY: SECURITY,QUALITY & EFFICIENCY. The International Archives of the Photogrammetry,Remote Sensing and Spatial Information Sciences. Vol. XXXVII. Part B1.Beijing 2008” describes experiments for covering larger areas with UAVsand generating an overview image.

There are some software tools for image stitching available. However,they have several restrictions/assumptions (camera position andorientation, distance to objects etc.). For example, AutoPano(http://www.autopano.net/) takes a set of images and generates anoverview (mosaick) which is visually most appealing. AutoPano stitchesfor beauty, at all areas where are less images or less overlap thedistortions are still high with AutoPano. Another tool is PTGui Pro(http://www.ptgui.com/). PTGui stitches most panoramas fullyautomatically, but at the same time provides full manual control overevery single parameter.

SUMMARY

According to an embodiment, an apparatus for generating an overviewimage of a plurality of images may have a storage unit configured tostore a plurality of processed images of the overview image, whereineach processed image of the plurality of processed images is assigned toa position of the overview image, wherein the storage unit is configuredto provide the overview image having the plurality of processed imagesat their assigned positions for displaying; and an image processorconfigured to determine feature points of a new image and configured tocompare the determined feature points of the new image with featurepoints of a stored processed image to identify common feature points andto acquire 3-dimensional positions of the common feature points, whereinthe image processor is configured to determine common feature pointslocated within a predefined maximum distance of relevance to a referenceplane based on the 3-dimensional positions of the common feature pointsto identify relevant common feature points, wherein the image processoris configured to process the new image by assigning the new image to aposition in the overview image based on a comparison of an imageinformation of each relevant common feature point of the new image withan image information of each corresponding relevant common feature pointof the stored processed image without considering common feature pointslocated beyond the predefined maximum distance of relevance to thereference plane, wherein the storage unit is configured to add theprocessed new image with the assigned position to the plurality ofprocessed images of the overview image.

According to another embodiment, a method for generating an overviewimage of a plurality of images may have the steps of storing a pluralityof processed images of the overview image, wherein each processed imageof the plurality of processed images is assigned to a position in theoverview image; determining feature points of a new image; comparing thedetermined feature points of the new image with feature points of astored processed image to identify common feature points and to acquire3-dimensional positions of the common feature points; determining commonfeature points located within a predefined maximum distance of relevanceto a reference plane based on the 3-dimensional positions of the commonfeature points to identify relevant common feature points; processing anew image by assigning the new image to a position in the overview imagebased on a comparison of an image information of each relevant commonfeature point of the new image with an image information of eachcorresponding relevant common feature point of the stored processedimage without considering common feature points located beyond thepredefined maximum distance of relevance to the reference plane; addingthe new image with the assigned position to the plurality of processedimages of the overview image; providing the overview image having theplurality of processed images at their assigned positions fordisplaying.

According to another embodiment, a computer program may have a programcode for performing the method for generating an overview image of aplurality of images, which may have the steps of storing a plurality ofprocessed images of the overview image, wherein each processed image ofthe plurality of processed images is assigned to a position in theoverview image; determining feature points of a new image; comparing thedetermined feature points of the new image with feature points of astored processed image to identify common feature points and to acquire3-dimensional positions of the common feature points; determining commonfeature points located within a predefined maximum distance of relevanceto a reference plane based on the 3-dimensional positions of the commonfeature points to identify relevant common feature points; processing anew image by assigning the new image to a position in the overview imagebased on a comparison of an image information of each relevant commonfeature point of the new image with an image information of eachcorresponding relevant common feature point of the stored processedimage without considering common feature points located beyond thepredefined maximum distance of relevance to the reference plane; addingthe new image with the assigned position to the plurality of processedimages of the overview image; providing the overview image having theplurality of processed images at their assigned positions fordisplaying, when the computer program runs on a computer or a microcontroller.

An embodiment of the invention provides an apparatus for generating anoverview image of a plurality of images. Each image of the plurality ofimages comprises associated meta-data. The apparatus comprises an imagepreprocessor, a storage unit and an image processor. The imagepreprocessor is configured to preprocess a new image by assigning thenew image to a position in the overview image based on a positioninformation contained by meta-data of the new image. Further, thestorage unit is configured to store a plurality of preprocessed orprocessed images. Each preprocessed or processed image of the pluralityof preprocessed or processed image is assigned to a position in theoverview image. Additionally, the storage unit is configured to providethe overview image containing the plurality of preprocessed or processedimages at their assigned positions for displaying. Further, the imageprocessor comprises an accuracy information input for receiving anaccuracy information of the position information and a controllableprocessing engine. The image processor is configured to determine anoverlap region of the preprocessed new image and a stored preprocessedor stored processed image within the overview image based on theassigned position of the new image and the assigned position of thestored preprocessed or stored processed image. Additionally, thecontrollable processing engine is configured to process the new image byre-adjusting the assigned position of the new image based on acomparison of features of the overlap region of the new image and thestored preprocessed or stored processed image. For this, thecontrollable processing engine is controlled by the accuracy informationof the position information received by the accuracy information input,so that a maximal re-adjustment of the assigned position of the newimage is limited based on the received accuracy information of theposition information.

Since the re-adjustment of the assigned position of the new image islimited based on the accuracy information of the position information,distances between points in different images can be preserved moreaccurate than with an image-based stitching algorithm withoutlimitation. In this way, a position-based generation of an overviewimage can be refined by an image-based re-adjustment without the risk oflosing completely the reference to real distances (the geo-reference) inthe overview image. Further, a very fast generation of an overview imagecontaining the new image may be possible, since a position of the newimage in the overview image is already assigned after preprocessing. Theprocessing of the new image for refining the overview image may be doneafterwards for improving the smoothness of transitions from the newimage to overlapping images for obtaining a nice-looking picture.

Another embodiment of the invention provides an apparatus for generatingan overview image of a plurality of images comprising a storage unit andan image processor. The storage unit is configured to store a pluralityof processed images of the overview image. Each image of the pluralityof images is assigned to a position in the overview image. Further, thestorage unit is configured to provide the overview image containing theplurality of processed images at their assigned positions fordisplaying. The image processor is configured to determine featurepoints of a new image and configured to compare the determined featurepoints of the new image with feature points of a stored processed imageto identify common feature points and to obtain 3-dimensional positionsof the common feature points. Further, the image processor is configuredto determine common feature points located within a predefined maximumdistance of relevance to a reference plane based on the 3-dimensionalpositions of the common feature points to identify relevant commonfeature points. Additionally, the image processor is configured toprocess the new image by assigning the new image to a position in theoverview image based on a comparison of an image information of eachrelevant common feature point of the new image with an image informationof each corresponding relevant common feature point of the stored imagewithout considering common feature points located beyond the predefinedmaximum distance to the reference plane. Further, the storage unit isconfigured to add the new processed image with the assigned position tothe plurality of images of the overview image.

Since only feature points located in or near the reference plane areconsidered for determining the position of the new image in the overviewimage, errors caused by considering reference points at largelydifferent elevations can be reduced and therefore the distance betweenpoints in different images can be preserved more accurate and/or thesmoothness of the transition between images can be increased to obtain anice-looking overview image.

Some embodiments according to the invention relate to an unmanned aerialvehicle comprising a camera, a sensor system, an accuracy determiner anda transmitter. The camera is configured to take an image during a flightof the unmanned aerial vehicle. The sensor system is configured todetermine a position of the unmanned aerial vehicle at the time theimage is taken to obtain a position information associated to the takenimage. Further, the accuracy determiner is configured to determine anaccuracy of the determination of the position of the unmanned aerialvehicle to obtain an accuracy information associated to the determinedposition information. The transmitter is configured to transmit thetaken image together with associated meta-data containing the positioninformation of the taken image and the accuracy information of theposition information.

By determining and transmitting an information of the accuracy of theposition information, the accuracy information can be taken intoaccount, for example, for the generation of an overview image containingthe taken image later on.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments according to the invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 is a block diagram of an apparatus for generating an overviewimage of a plurality of images;

FIG. 2 is a block diagram of an apparatus for generating an overviewimage of a plurality of images;

FIG. 3 is a schematic illustration of position and orientation errors ofunmanned aerial vehicles;

FIG. 4 is a schematic illustration of the generation of an overviewimage of a plurality of images taken by an unmanned aerial vehicle;

FIG. 5 is a flowchart of a method for generating an overview image of aplurality of images;

FIG. 6 is a block diagram of an apparatus for generating an overviewimage of a plurality of images;

FIG. 7 is a flowchart of a method for generating an overview image of aplurality of images;

FIG. 8 is a flowchart of a method for generating an overview image of aplurality of images;

FIG. 9 is a block diagram of an unmanned aerial vehicle;

FIG. 10 a is an example for an overview image of a plurality of images;

FIG. 10 b is another example for an overview image of a plurality ofimages;

FIG. 11 a is a diagram indicating a comparison between correlation ofthe overlapping parts of two adjacent images in different approaches;

FIG. 11 b is a diagram indicating the relative distance between theestimated position and the GPS position;

FIG. 12 is an illustration of a position-based alignment of eightimages;

FIG. 13 is an illustration of a position and orientation-based alignmentof eight images; and

FIG. 14 is an illustration of an image-based alignment using SIFTfeatures.

DETAILED DESCRIPTION OF THE INVENTION

In the following, the same reference numerals are partly used forobjects and functional units having the same or similar functionalproperties and the description thereof with regard to a figure shallapply also to other figures in order to reduce redundancy in thedescription of the embodiments.

FIG. 1 shows a block diagram of an apparatus 100 for generating anoverview image 124 of a plurality of images according to an embodimentof the invention. Each image of the plurality of images comprisesassociated meta-data. The apparatus 100 comprises an image preprocessor110, a storage unit 120 and an image processor 130. The imagepreprocessor 110 is connected to the image processor 130 and the imageprocessor 130 is connected to the storage unit 120. The imagepreprocessor 110 preprocesses a new image 102 by assigning the new image102 to a position in the overview image 124 based on a positioninformation contained by the meta-data of the new image 102. Further,the storage unit 120 stores a plurality of preprocessed or processedimages 112, 122, wherein each preprocessed or processed image 112, 122of the plurality of preprocessed or processed images comprises anassigned position in the overview image 124. Additionally, the storageunit 120 is able to provide the overview image 124 containing theplurality of preprocessed or processed images at their assignedpositions for displaying. The image processor 130 comprises an accuracyinformation input 134 for receiving an accuracy information 104 of theposition information and a controllable processing engine 132. Theaccuracy information input 134 is connected to the controllableprocessing engine 132 and the controllable processing engine 132 isconnected to the image preprocessor 110 and the storage unit 120.Further, the image processor 130 determines an overlap region of thepreprocessed new image 112 and a stored preprocessed or stored processedimage 112, 122 within the overview image 124 based on the assignedposition of the new image 112 and the assigned position of the storedpreprocessed or stored processed image 112, 122. The controllableprocessing engine 132 processes the preprocessed new image 112 byre-adjusting the assigned position of the preprocessed new image 112based on a comparison of features of the overlap region of thepreprocessed new image 112 and features of the overlap region of thestored preprocessed or stored processed image 112, 122. For this, thecontrollable processing engine is controlled by the accuracy informationof the position information 104 received by the accuracy informationinput 134, so that a maximal re-adjustment of the assigned position ofthe preprocessed new image 112 is limited based on the received accuracyinformation 104 of the position information. Additionally, the storageunit 120 adds the processed new image 122 with the re-adjusted assignedposition to the plurality of preprocessed or processed images of theoverview image 124.

Since the re-adjustment of the assigned position of the preprocessed newimage 112 is limited based on the received accuracy information 104 ofthe position information, the image-based processing of the preprocessedimage 112 and the overlapping stored preprocessed or processed image112, 122 preserves the connection of the distances of points indifferent images of the overview image 124 to the distances in realityat least with the accuracy of the position information contained by themeta-data. In this way, a position-based alignment of images in anoverview image can be refined by an image-based alignment without losingthe whole information about distances between points in the overviewimage. So, the accuracy of preserving the distance in the overview imagecan be significantly increased. Further, the smoothness of transitionsof images in the overview image may be increased in comparison to onlyposition-based or orientation-based alignments.

The new image 102 may be provided, for example, from a storage unit(e.g. a memory card of a digital camera or a storage device of acomputer) or may be transmitted online from an unmanned aerial vehicleor an airplane taking images of an area.

Each image comprises associated meta data, which is stored together withthe image or transmitted together with the image, for example. Themeta-data contains a position information of the image, but may alsocontain further data (e.g. roll, pitch, yaw of a camera, the image istaken with). The position information may be, for example, a GPSposition (global positioning system) or another position definitionindicating a relative or an absolute position allowing to assign the newimage to a position in the overview image.

The storage unit 120 stores a plurality of preprocessed or processedimages of the overview image. In this connection, a preprocessed image112 is an image comprising a position in the overview image assigned bythe image preprocessor 110 before the assigned position is re-adjustedby the image processor 130. Consequently, a processed image 122 is animage after re-adjustment of the assigned position by the imageprocessor 130.

For example, a preprocessed image 112 may be stored directly afterpreprocessing by the storage unit 120 to enable the storage unit 120 toprovide an overview image containing the new image very fast.Afterwards, the new image may be processed by the image processor 130 torefine the assigned position of the preprocessed new image 112.Alternatively, the preprocessed image 112 is directly processed by theimage processor 130 without storing the preprocessed image 112.

The storage unit 120 is able to provide the overview image 124 to adisplay (which may be an optional part of the apparatus 100).

The image processor 130 determines an overlap region of the preprocessednew image 112 and a stored preprocessed or stored processed image 122.Although it may be possible that none of the already stored preprocessedor stored processed images 112, 122 overlap with the preprocessed newimage 112, it is getting more likely when the number of images alreadycontained by the overview image stored by the storage unit 120increases. If a preprocessed new image 112 does not comprise an overlapregion with any of the other images 112, 122 of the overview image 124already stored by the storage unit 120, the preprocessed new image 112may be stored directly by the storage unit 120.

The controllable processing engine 132 processes the preprocessed newimage 112 by re-adjusting the assigned position, wherein there-adjustment is limited based on the accuracy information 104 of theposition information. The accuracy information 104 of the positioninformation may be the same for all images, may be updated periodicallyor depending on a change of the accuracy of the position information.Alternatively, each image may comprise an individual accuracyinformation 104 of the individual position information contained by theassociated meta-data. For example, if the images are taken by anunmanned aerial vehicle, the unmanned aerial vehicle may determine aposition and an accuracy of the position at the time a picture is takenand transmits this information together with the image to the apparatus100. Alternatively, the unmanned aerial vehicle may transmit accuracyinformation periodically or when the accuracy changes more than apredefined threshold. In another example, the accuracy of the positioninformation is known and stored by a storage device, which provides theaccuracy information of the position information to the accuracyinformation input 134. By limiting the re-adjustment of the assignedposition by the accuracy information 104 of the position information itmay be guaranteed that the loss of accuracy of the connection betweendistances of points in the overview image to distances in reality is notlarger than the already existent inaccuracy due to the limited accuracyof the position information.

The re-adjustment is done based on a comparison of features of theoverlap region of the preprocessed new image 112 and the storedpreprocessed or stored processed image 112, 122. For example, thefeatures of the overlap region may be common feature points identifiedin the overlap regions of both images or areas around feature points ofthe overlap regions of images. Although only a re-adjustment of theassigned position of the preprocessed new image 112 is mentioned, at thesame time a re-adjustment (or a further re-adjustment) of the assignedposition of the stored preprocessed or stored processed image 112, 122may be done. In this case, the re-adjustment of the assigned position ofthe stored preprocessed or stored processed image 112, 122 may also belimited by an accuracy information of the position information of thestored preprocessed or stored processed image 112, 122.

The image preprocessor 110, the storage unit 120 and the image processor130 may be, for example, independent hardware units or part of acomputer or micro controller as well as a computer program or a softwareproduct configured to run on a computer or micro controller.

Additionally to the assignment of the new image 102 to the position inthe overview image 124, the image preprocessor 110 may preprocess thenew image 102 by considering a roll information, a pitch informationand/or a yaw information to correct an orientation and/or a perspectivedistortion of the new image 102. For example, the new image 102 may betaken from an unmanned aerial vehicle. Such an unmanned aerial vehicleis usually small and therefore susceptible for wind and otherenvironmental influences. Therefore, sensors (IMU, inertial measurementunit) of the unmanned aerial vehicle may determine a roll, a pitch and ayaw of the unmanned aerial vehicle (or of the camera) at the time animage is taken and add a roll information, a pitch information and/or ayaw information to the meta-data of the image. The image preprocessor110 can use this information contained by the meta-data to compensate anorientation offset or a perspective distortion of the new image 102.

Further, the controllable processing engine 132 may process thepreprocessed new image 112 by re-adjusting an orientation or aperspective distortion based on the comparison of the features of theoverlap region. In this case, the controllable process engine may befurther controlled by accuracy information 104 of the roll information,the pitch information or the yaw information received by the accuracyinformation input 134, so that the maximal re-adjustment of the assignedposition of the preprocessed new image 112 and a maximal re-adjustmentof the orientation or the perspective distortion is limited based on theaccuracy information 104 of the roll information, the pitch informationor the yaw information. In other words, the maximal re-adjustment of theassigned position may be limited by the accuracy of the determination ofthe position, which may also depend on the accuracy of the roll, thepitch and/or the yaw. In this way, for example, the position andorientation of the camera, the image was taken with, an unmanned aerialvehicle or another vehicle taking that new image may be consideredduring re-adjustment of the assigned position of the new image.Additionally, also an orientation or a perspective distortion may bere-adjusted and limited based on the accuracy information 104.

FIG. 3 shows an example for position and orientation errors, which canbe taken into account for the re-adjustment of the position of the newimage 112. It indicates the GPS error range (the real position is inthis range) and the tilting error range. The sum of these two errors maygive the total positioning error, which may be represented by theaccuracy information of the position information. It shows an examplefor a maximum position error (accuracy of the position information), ifthe new image 102 is taken by an unmanned aerial vehicle.

In some embodiments according to the invention, the image preprocessor110 may receive successively a plurality of new images 102 andpreprocesses each received new image 102. Further, the storage unit 120may store each preprocessed new image 112 and provides an updatedoverview image 124 after storing a predefined number of preprocessed newimages 112. The predefined number of images may be one, so that thestorage unit 120 may provide an updated overview image after storingeach preprocessed new image 112. Alternatively, the predefined number ofpreprocessed images may be higher than one, so that the overview imageis updated periodically after storing the predefined number ofpreprocessed images. In this way, an overview image 124 can be providedvery fast, since the position and/or orientation-based preprocessing canbe done significantly faster than the image-based processing by theimage processor 130. For example, the apparatus 100 may receivecontinuously new images from an unmanned aerial vehicle, which can beprovided for displaying in an overview image directly afterpreprocessing. A re-adjustment of the assigned position by the imageprocessor 130 for refinement of the transition between overlappingimages may be done later on.

FIG. 2 shows a block diagram of an apparatus 200 for generating anoverview image 124 of a plurality of images according to an embodimentof the invention. The apparatus 200 is similar to the apparatus shown inFIG. 1, but comprises additionally an optional unmanned aerial vehiclecontrol unit 240 connected to the storage unit 120 and the imagepreprocessor 110 is connected to the storage unit 120.

In this way, the storage unit 120 may add the preprocessed new image 112with the assigned position to the plurality of preprocessed or processedimages. Further, the storage unit 120 may provide the overview image 124containing the preprocessed new image 112 at the assigned position fordisplaying before the preprocessed new image 112 is processed by theimage processor 130 as already mentioned above. In this way, an overviewimage 124 already containing the preprocessed new image 112 can beprovided very fast. Afterwards, the accuracy of the distance between twopoints in different images and/or the smoothness of the transitionsbetween the preprocessed new image 112 and other overlapping images ofthe plurality of preprocessed or processed images 112, 122 may beincreased by re-adjusting the assigned position of the preprocessed newimage 112 by the image processor 130. In other words, afterpreprocessing the new image 112, the storage unit 120 may provide theoverview image 124 and later the storage unit 120 may provide a refinedoverview image 124 after processing the preprocessed image 112 by theimage processor 130.

If the preprocessed new image 112 is stored by the storage unit 120, thestorage unit 120 may add the processed new image 122 by replacing thestored preprocessed new image 112 by the processed new image 122.

Optionally, the apparatus 200 may comprise an unmanned aerial vehiclecontrol unit 240. The unmanned aerial control unit 240 may generate acontrol signal 242 for piloting an unmanned aerial vehicle to a regioncorresponding to an area of the overview image 124 being not alreadycovered by an image of the plurality of preprocessed or processed images112, 122. In other words, if an unmanned aerial vehicle is used fortaking images of an area from which an overview image should begenerated, then the unmanned aerial vehicle control unit 240 may pilotthe unmanned aerial vehicle to uncovered areas of the overview image toobtain images from the uncovered areas.

Fittingly, FIG. 4 shows an example of an apparatus 200 piloting anunmanned aerial vehicle 460 with a camera 462 for taking images of thearea below the unmanned aerial vehicle 460. The apparatus 200 maytransmit a control signal to the unmanned aerial vehicle 460 forpiloting the unmanned aerial vehicle 460 to uncovered regions of theoverview image 452. The overview image 452 may be provided to a display450. In this example, the display 450 shows covered areas 454 anduncovered areas of the overview image 452. The unmanned aerial vehicle460 may take images from the uncovered areas and provide the image dataas well as corresponding meta-data (e.g. position information, rollinformation, pitch information and/or yaw information) to the apparatus200 for generating the overview image 452 of the plurality of images454.

FIG. 5 shows a flowchart of a method 500 for generating an overviewimage of a plurality of images according to an embodiment of theinvention. Each image of the plurality of images comprises associatedmeta-data. The method 500 comprises preprocessing 510 a new image,storing 520 a plurality of preprocessed or processed images of theoverview image, determining 530 an overlap region of the preprocessednew image and a stored preprocessed or stored processed image within theoverview image, processing 540 the preprocessed new image, adding 550the processed new image to the plurality of preprocessed or processedimages and providing 560 the overview image containing the plurality ofpreprocessed or processed images. The new image is preprocessed 510 byassigning the new image to a position in the overview image based on aposition information being contained by the meta-data of the new image.Further, each stored preprocessed or stored processed image of theplurality of preprocessed or processed images comprising an assignedposition in the overview image. The overlap region of the preprocessednew image and the stored preprocessed or stored processed image withinthe overview image is determined 530 based on the assigned position ofthe preprocessed new image and the assigned position of the storedpreprocessed or stored processed image. Further, the processing 540 ofthe preprocessed new image is done by re-adjusting the assigned positionof the preprocessed new image based on a comparison of features of theoverlap region of the preprocessed new image and the stored preprocessedor stored processed image. The processed new image is added 550 to theplurality of preprocessed or processed images with the re-adjustedassigned position. Further, the overview image is provided 560containing the plurality of preprocessed or processed images at theirassigned positions for displaying.

Further, the method 500 may comprise adding the preprocessed new imageto the plurality of preprocessed or processed images of the overviewimage, displaying the overview image containing a preprocessed new imageat the assigned position and displaying the overview image containingthe processed new image at the re-adjusted assigned position. In thisway, a rough overview image may be displayed very fast and a refinedoverview image may be displayed later on.

Additionally, the method 500 may comprise further steps representing theoptional features of the described concept mentioned above.

FIG. 6 shows a block diagram of a apparatus 600 for generating anoverview image 614 of a plurality of images according to an embodimentof the invention. The apparatus 600 comprises a storage unit 610 and animage processor 620. The storage unit 610 is connected to the imageprocessor 620. The storage unit 610 stores a plurality of processedimages of the overview image 614. Each processed image 612 of theplurality of processed images comprises an assigned position in theoverview image 614. Further, the storage unit 610 is able to provide theoverview image 614 containing the plurality of processed images at theirassigned positions for displaying. The image processor 620 determinesfeature points of a new image 602 and compares the determined featurepoints of the new image 602 with feature points of a stored processedimage 612 to identify common feature points and to obtain 3-dimensionalpositions of the common feature points. Further, the image processor 620determines common feature points located within a predefined maximumdistance of relevance to a reference plane based on the 3-dimensionalpositions of the common feature points to identify relevant commonfeature points. Additionally, the image processor 620 processes the newimage 602 by assigning the new image to a position in the overview image614 based on a comparison of an image information of each relevantcommon feature point of the new image 602 with an image information ofeach corresponding relevant common feature point of the stored processedimage 612 without considering common feature points located beyond thepredefined maximum distance to the reference plane. Further, the storageunit 610 adds the processed new image 612 with the assigned position tothe plurality of processed images of the overview image 614.

By taking into account only common feature points of the new image 602and the stored processed image 612 being located within a predefinedmaximum distance of relevance to a reference plane, a falsification ofthe assigned position of the new image in the overview image 614 bycommon feature points located at significantly different elevationlevels than the relevant common feature points may be reduced. In thisway, distances between two points in different images may be preservedmore accurate and/or the smoothness of transitions between overlappingimages may be increased.

The storage unit 610 may store each processed image together with theassigned position in the overview image 614. The overview image 614 maybe provided to display (which may be an optional part of the apparatus600).

The feature points determined by the image processor may be, forexample, corners or walls of houses, the contours of a car, roadmarkings or similar features. These feature points may be determined byvarious algorithms as, for example, SIFT (scale-invariant featuretransform) or SURF (speeded up robust features). After processing thenew image 602 by the image processor 620, the storage unit 610 may addthe processed new image 612 together with the determined feature pointsof the new image to the plurality of processed images. In this way, thefeature points determination for an image may only be needed one time.

After determining the feature points of the new image 602, these featurepoints may be compared to feature points of a stored processed image(e.g. an overlapping image) to identify common feature points by using,for example, a correlation function (e.g. nearest neighbor). Further,for these common feature points 3-dimensional positions of the commonfeature points may be determined, for example, by using methods ofmulti-view geometry.

Then, common feature points located within a predefined maximum distanceof relevance to a reference plane are determined. The reference planemay be predefined or may be determined by the image processor 620 basedon the 3-dimensional positions of the common feature points. Forexample, a predefined reference plane may be a horizontal plane of a3-dimensional coordinate system (e.g. Cartesian coordinate system) withthe z-coordinate equal to 0 or another predefined value. Alternatively,the image processor 620 may determine the reference plane by fitting ahorizontal plane to the 3-dimensional positions of the common featurepoints, so that a maximal number of common feature points are locatedwithin a predefined maximal fitting distance to the reference plane orso that the reference plane is as far away as possible from a cameraposition, the new image was taken from, with at least one common featurepoint located within the predefined maximal fitting distance. Themaximal fitting distance may vary between zero and, for example, a valuedepending on a maximal height difference of the common feature points.Alternatively, the reference plane may be a non-horizontal plane.

By choosing the maximal predefined distance of relevance, the accuracyof preserving the distances between points in different images and/orthe smoothness of transitions between images may be influenced. Bychoosing a large maximum distance of relevance, more relevant commonfeature points are identified, which are taken into account forassigning the position of the new image. This may increase thestatistic, since more common feature points are considered, but may alsoincrease the error obtained by considering points at different elevationlevels. An opposite effect is obtained by choosing a low predefinedmaximum distance of relevance. The predefined maximum distance ofrelevance may also be zero, so that only common feature points areidentified as relevant which are located in the reference plane. Themaximal fitting distance may be equal to the maximal distance ofrelevance.

For example, common feature points may be road markings and thereference plane may be the road. In this way, only common feature pointslocated at the height of the street may be considered for assigning theposition. In this way, an error obtained by considering feature pointsat different elevation levels (e.g. feature points located at cars,houses or street lights) may be suppressed completely or nearlycompletely.

The image information of a relevant common feature point may be, forexample, the position of the feature point in the image itself or theimage data of an area of the image around the relevant common featurepoint. More accurately, the image information of a relevant commonfeature point in the new image 602 may be the position of the relevantcommon feature point in the new image 602 and the image information of arelevant common feature point in the stored image 612 may be a positionof the relevant common feature point in the stored image. With this, theimage processor 620 may assign the new image 602 to a position in theoverview image 614, so that a distance between a position of a relevantcommon feature point in the new image 602 and a position of acorresponding relevant common feature point in the stored image 612 isminimized or so that a sum of distances between the position of therelevant common feature points in the new image 602 and the positions ofthe respective corresponding relevant common feature points in thestored image 612 is minimized.

Alternatively, the image information of a relevant common feature pointin the new image 602 may be an area of the new image 602 around therelevant common feature point of a predefined size and the imageinformation of a relevant common feature point in the stored image 612may be an area of the stored image 612 around a relevant common featurepoint of the predefined size. In this case, the image processor 620 mayassign the new image 602 to a position in the overview image 614, sothat a correlation value of an area around a relevant common featurepoint of the new image 602 and an area around a corresponding relevantcommon feature point of the stored image 612 is maximized or so that thesum of correlation values of areas around relevant common feature pointsof the new image 602 and areas around respective corresponding relevantcommon feature points of the stored image 612 is maximized. Acorrelation value may be determined, for example, based on a givencorrelation function for correlating images or parts of images.

Further, both mentioned examples may be combined so that the imageprocessor 620 may assign the new image 602 to a position in the overviewimage 614 based on the minimization of the distance or the sum ofdistances and re-adjusts the assignment of the new image 602 to aposition in the overview image 614 based on the maximization of acorrelation value or a sum of correlation values.

The image processor 620 may determine feature points of the new image602 for the whole image or only parts of the new images 602 overlappinganother image of the plurality of processed images. Determining featurepoints from the whole image may increase the efforts, but the determinedfeature points can be stored together with the new image afterprocessing, so that the feature points may only be determined once.

Independent from the determination of the feature points, only featurepoints located within an overlapping region of the new image and thestored image 612 may be considered for identifying common featurepoints. For this, for example the image processor 620 may determine anoverlap region of the new image 602 and the stored image 612. Further,the image processor 620 may compare determined feature points of the newimage 602 located in the overlap region with feature points of thestored image 612 located in the overlap region, while determined featurepoints outside the overlap region are neglected for the identificationof common feature points.

The storage unit 610 and the image processor 620 may be, for example,independent hardware units or part of a computer or micro controller aswell as a computer program or a software product configured to run on acomputer or a micro controller.

FIG. 7 shows a flowchart of a method 700 for generating an overviewimage of a plurality of images according to an embodiment of theinvention. The method 700 comprises storing 710 a plurality of processedimages, determining 720 feature points of a new image, comparing 730 thedetermined feature points of the new image with feature points of astored processed image, determining 740 common feature points,processing 750 the new image, adding 760 the processed new image to theplurality of processed images of the overview image and providing 770the overview image. Each stored processed image of the plurality ofprocessed images comprises an assigned position in the overview image.The determined feature points of the new image are compared 730 withfeature points of a stored processed image to identify common featurepoints and to obtain 3-dimensional positions of the common featurepoints. Further, common feature points located within a predefinedmaximum distance of relevance to a reference plane are determined 740based on the 3-dimensional positions of the common feature points toidentify relevant common feature points. The new image is processed 750by assigning the new image to a position in the overview image based ona comparison of an image information of each relevant common featurepoint of the new image with an image information of each correspondingrelevant common feature point of the stored processed image withoutconsidering common feature points located beyond the predefined maximumdistance of relevance to the reference plane. Further, the processed newimage is added 760 to the plurality of processed images of the overviewimage with the assigned position. Additionally, the overview imagecontaining the plurality of processed images at their assigned positionsis provided 770 for displaying.

Optionally, the method 700 may comprise further steps representingfeatures of the described concept mentioned above.

Some embodiments according to the invention relate to an apparatus forgenerating an overview image of a plurality of images combining thefeatures of the apparatus shown in FIG. 1 and the features of theapparatus shown in FIG. 6.

For example, the image processor 130 of apparatus 100 may re-adjust theassigned position of the preprocessed new image based on a comparison ofan image information of relevant common feature points of the new imagewith an image information of each corresponding relevant common featurepoint of a stored processed image without considering common featurepoints located beyond a predefined maximum distance of relevance to areference plane. For this, the image processor 130 may determine featurepoints of the preprocessed new image and may compare the determinedfeature points of the preprocessed new image with feature points of astored processed image to identify common feature points and to obtain3-dimensional positions of the common feature points. Further, the imageprocessor 130 may determine common feature points located within apredefined maximum distance of relevance to a reference plane based onthe 3-dimensional positions of the common feature points to identifyrelevant common feature points.

Optionally, the apparatus 100 may realize further features, for example,mentioned in connection with the apparatus 600 shown in FIG. 6.

The other way around, for example, the apparatus 600 may compriseadditionally an image preprocessor 110 and the image processor 620 maycomprise an accuracy information input 134 and a controllable processingengine 132 as mentioned in connection with the apparatus 100 shown inFIG. 1. In this example, each image of the plurality of images maycomprise associated meta-data. Further, the image preprocessor 110 maypreprocess a new image by assigning the new image to a position in theoverview image based on a position information contained by themeta-data of the new image. The storage unit 610 may store a pluralityof preprocessed or processed images of the overview image, wherein eachpreprocessed or processed image of the plurality of preprocessed orprocessed images comprises an assigned position in the overview image.The storage unit 610 may provide an overview image containing theplurality of preprocessed or processed images at their assignedpositions for displaying. Further, the image processor 620 may determinean overlap region of the preprocessed new image and a storedpreprocessed or stored processed image within the overview image basedon the assigned position of the preprocessed new image and the assignedposition of the stored preprocessed or stored processed image. Thecontrollable processing engine 132 may process the preprocessed image byre-adjusting the assigned position of the preprocessed new imageaccording to the originally defined image processor 610, but with therestriction that the controllable processing engine 132, and in that waythe image processor 610, is controlled by the accuracy information ofthe position information received by the accuracy information input, sothat a maximal re-adjustment of the assigned position of thepreprocessed new image is limited based on the received accuracyinformation of the position information.

Optionally, the apparatus 600 may realize further, for example, featuresmentioned in connection with apparatus 100 shown in FIG. 1 or apparatus200 shown in FIG. 2.

In the following, a detailed example for using a combination of thefeatures described above is given. In this example, the images are takenby an unmanned aerial vehicle, although the described inventive conceptmay also be used or implemented, if images are provided, for example, bya storage unit (e.g. memory card or multimedia card of a digital cameraor a hard disk of a computer).

A very simple and naive approach is to align the images based on thecamera's position. Hence, for image alignment the world coordinates ofthe camera are mapped to corresponding pixel coordinates in thegenerated overview image. Defining the origin of the overview image ofthe observed target area as ∘_(world)=(lat; lon; alt)^(T) in worldcoordinates, all image coordinates are related to this origin on thelocal tangential plane (LTP) by approximation to the earth model WGS84.Given the camera's position area covered by the picture in worldcoordinates relative to the origin is computed taking into account thecamera's intrinsic parameters. The relative world coordinates aredirectly related to the pixel coordinates in the generated overviewimage. An example of the resulting overview image is depicted in FIG. 12utilizing the placement function with transformation being just a simpletranslation for each image. In this approach reasonably accurateposition information is assumed and a nadir view but do not take intoaccount the camera's orientation. Obviously, effects introduced bynon-planar surfaces can not be compensated with this approach. A moreadvanced approach is to extend the naive position-based alignment bycompensating the camera's orientation deviation (i.e., roll, pitch, yawangles). The placement function of the individual images to generate theoverview image is the same as before. But instead of considering onlytranslation, a perspective transformation with eight degrees of freedommay be used.

If a nadir view (i.e., neglecting deviation of roll and pitch angles) isassumed, the transformation is reduced to a similarity transformation.An example is shown in FIG. 13.

Further, image-based alignment can be categorized into pixel-based, andfeature-based methods. The idea is to find transformations T_(i) andconsequently the position of each new image which maximizes the qualityfunction:

λ(Merge(I _(res,i−1) ,T _(i)(I _(i))))

The pixel-based approaches are computationally more expensive becausethe quality function is computed from all pixels in the overlappingparts of two images. Feature-based approaches try to reduce thecomputational effort by first extracting distinctive feature points andthen match the feature points in overlapping parts. Depending on thechosen degree of freedom the resulting transformation ranges from asimilarity transformation to a perspective transformation. The benefitof this approach is that the generated overview image is visually moreappealing. But on the other hand, the major disadvantages are that thesearch space grows with the number of images to be stitched and theimages may get distorted. An example is shown in FIG. 14.

According to one aspect multiple small-scale UAVs may be deployed tosupport first responders in disaster assessment and disaster management.In particular commercially available quadrocopters may be used sincethey are agile, easy to fly and very stable in the air due tosophisticated on-board control. The UAV may be equipped with an RGBcamera (red, green blue).

The intended use-case can be sketched as follows: The operator firstspecifies the areas to be observed on a digital map and defines thequality parameters for each area (see for example “M. Quaritsch, E.Stojanovski, C. Bettstetter, G. Friedrich, H. Hellwagner, M. Hofbaur, M.Shah, and B. Rinner. Collaborative Microdrones: Applications andResearch Challenges. In Proceedings of the Second InternationalConference on Autonomic Computing and Communication Systems (Autonomics2008), page 7, Turin, Italy, September 2008”). Quality parametersinclude, for example, the spatial and temporal resolution of thegenerated overview image, and the minimum and maximum flight altitude,among others.

Based on the user's input, the system generates plans for the individualdrones to cover the observation areas (see for example “M. Quaritsch, K.Kruggl, D. Wischounig-Strucl, S. Bhattacharya, M. Shah, and B. Rinner.Networked UAVs as Aerial Sensor Network for Disaster ManagementApplications. e&i Journal, 127(3):56-63, March 2010”). Therefore, theobservation areas are partitioned into smaller areas covered by a singlepicture taken from a UAV flying at a certain height. The partitioninghas to consider a certain overlap of neighboring images which is neededby the stitching process. Given a partitioning the continuous areas tobe covered can be discretized to a set of so-called picture-points. Thepicture-points are placed in the center of each partition at the chosenheight. The pictures are taken with the camera pointing downwards (nadirview).

The mission planner component (e.g. unmanned aerial vehicle controlunit) generates routes for individual UAVs such that each picture-pointis visited taking into account the UAV's resource limitations. Theimages together with metadata (i.e., the position and orientation of thecamera) are transferred to the base-station during flight where theindividual images are stitched to an overview image.

The major goal is to generate an overall image I_(res;n) of the targetarea given a set of n individual images {I_(i)}. For example, theoverall image can be iteratively constructed as follows:

I _(res,0) =O,I _(res,i)=Merge(I _(res,i−1) ,T _(i)(I _(i)))

where O is an empty background matrix, T is a transformation functionand the Merge function combines the transformed image to the overallimage.

This mosaicking can be described as an optimization problem, in whichT_(i) has to be found in a way that it maximizes a quality functionλ(I_(res,i)). This quality function, based on the system use case,balances the visual appearance (improving the smoothness of transitions)and the geo-referencing accuracy (accuracy of preserving distances).While in some applications it is more important to have a visuallyappealing overview image, other applications may need accurategeo-referencing in the overview image. A quality function may be usedthat is a combination of the correlation between overlapping images andrelative distances in the generated overview image compared to theground truth.

Some challenges for solving the problem using images from low-flying,small-scale UAVs are mentioned in the following.

When taking images from a low altitude the assumption of a planarsurface is no longer true. Objects such as buildings, trees and evencars cause high perspective distortions in images. Without a commonground plane, the matching of overlapping images needs depthinformation. Image transformations exploiting correspondences of pointsat different elevations may result in severe matching errors.

Further, due to their light weight small-scale UAVs are vulnerable towind influences requiring high-dynamic control actions to achieve astable flight behavior. Even if the onboard camera position is activelycompensated, a perfect nadir-view of the images cannot be provided.

The UAV's auxiliary sensors such as GPS, IMU and altimeter are used todetermine its position and orientation. However, such auxiliary sensorsin small-scale UAVs provide only limited accuracy which is notcomparable with larger aircrafts. As consequence, it can not be reliedon accurate and reliable position, orientation and altitude data of theUAV. Hence it has to be dealt with this inaccuracy in the mosaickingprocess.

Additionally, the resources such as computation power and memoryon-board the UAVs but also on the ground station may be very limited. Indisaster situations it is usually not possible to have a huge computinginfrastructure available. The base-station may consist of notebooks andstandard desktop PCs. But at the same time, the overview image should bepresented as quick as possible

The individual images may be taken from multiple UAVs in an arbitraryorder. An incremental approach may be used to present the user theavailable image data as early as possible while the UAVs are still ontheir mission. The more images are taken the better the overview imagegets. This also means that a new image may need to adjust the positionof already processed image to improve the overall quality.

For example, as described an appropriate transformation T_(i) for eachimage I_(i) captured at a picture-point may be found in order to solvethe mosaicking problem. There are two basic approaches for computingthese transformations: The metadata approach exploits auxiliary sensorinformation to derive the position and orientation of the camera whichis then used to compute the transformations. In this case that auxiliarysensor data (i.e., GPS, altitude and time) may be provided for eachcaptured image. The image-based approach only exploits image data tocompute the transformations.

The proposed concept may realize a combination of metadata-based andimage-based methods enhancing the metadata-based alignment withimage-based alignment. The presented approaches vary in their resourcerequirements and their achieved results.

An aspect is to first place the new images based on the camera'sposition (position information) and orientation information (e.g. roll,nick, yaw) on the already generated overview image. In the next step,image-based methods are used to correct for inaccurate position andorientation information and at the same time improve the visualappearance. Since the approximate position of the image is already knownfrom the camera's position, the search-space can be significantlyreduced (by limiting the maximal re-adjustment). Thus, thetransformation T_(i) mentioned before may be split into twotransformation whereas the T_(i,pos) represents the transformation basedon the camera's position and orientation and T_(i,img) represents thetransformation which optimizes the alignment using the image-basedmethod.

Transformations T_(i,img) and T_(i,pos) which maximize the qualityfunction may be advantageous:

λ(Merge(I _(res,i−1) ,T _(i,img) ∘T _(i,pos)(I _(i)))),

The search space may be limited to a reduced set of possible positionsbased on the expected inaccuracy of position and orientation information(accuracy information of the position information and optionally theaccuracy information of the roll, nick or yaw).

With this proposed approach an appealing overview image withoutsignificant perspective distortions may be generated and at the sametime the relative distances and geo-references in the overview image canbe maintained. Moreover, this approach can cope with inaccurate positionand orientation information of the camera and thus avoid stitchingdisparities in the overview image.

In the following, some examples for technical details on imagemosaicking from micro-UAVs flying at low altitude are described. Apossible process 800 of aligning images taken from mico-UAVs flying atlow altitude is sketched in FIG. 8.

The input for the whole processing pipeline is a new image taken fromthe camera on-board the UAV and a set of meta-data. The meta-datacontains information on the (e.g. GPS-) position where the photo hasbeen taken as well as information on the UAV's and thus the camera'spose (yaw, nick, roll)—also known as extrinsic camera parameters. Due tothe limited capabilities of the UAVs—and the small and relatively cheapinertial measurement units (IMU)—the meta-data is inaccurate to somedegree. For example the accuracy of GPS-positions is in the range of upto 10 meters and also the measured angles have errors.

Firstly, the image is preprocessed 810. The image preprocessing 810 mayinclude several steps to improve the original images taken by a camera.For example, lens distortions are corrected.

Secondly, an image transformation based on meta-data is performed 820(assigning the new image to a position in the overview image). Given theimage's meta-data, quickly the transformation T_(i,pos) can be computedwhich is solely based on the meta-data. This transformation basicallyincludes a translation and rotation (to take into account the UAV'sheading(yaw)) and optionally warps the image as it has been taken at anadir view (i.e., correct nick and roll angles).

Thirdly, intersecting images may be found 830 (determining an overlapregion). Based on the image position in the pool of already processedimages those that intersect with the current one are identified. If noimage can be found, further processing is postponed until at least oneintersecting image is available. In this case the current image isplaced in the overview image only based on the meta-data (T_(i,pos)).

Fourthly, a (rough) 3D structure is computed 840. For this, the originalimage (new image) and the set of images that intersect (overlap) theoriginal one are considered. First feature-points from the new image areextracted 842 (determine feature points). Different algorithms can beused to compute feature-points (e.g., SIFT, SURF, . . . ). For thealready processed images the feature-points can be stored separately andthus avoid repeated computation. Consequently, a set of feature-pointsfor each image to consider is obtained. Next, the feature-points fromthe individual images are matched 844 (determine common feature points)using some correlation function (e.g., nearest neighbor). Thus, amapping, which feature-point in one image is (expected to be) the samepoint in the other images is established. Using this set ofcorresponding feature-points, for example, methods of multi-viewgeometry (structure from motion) can be exploited to estimate both, thecamera's positions and the 3D position of the feature-points 846.

Fifthly, the transformation is refined 850 based on the 3D structure andimage data. The input for this step is the set of intersecting imagesand the rough 3D structure of the scene computed in the previous step.By rough 3D structure, a point-cloud in 3D space with theircorresponding image coordinates is understood. One major goal is tomaintain geo-referencing as good as possible and at the same timecompute a visually appealing overview image. In order to achieve thisgoal, for example, the transformation is computed based on point-matcheson a common plane (reference plane, e.g. advantageously the groundplane) and ignore regions which are at different elevation levels (e.g.,cars, trees, etc.). For this, given the 3D point-cloud a common planemay be extracted. Thus horizontal planes may be fit 852 into thepoint-cloud and select a common plane (e.g., the plane far away from thecamera, or the plane with the highest number of points lying on it,etc.). For the next steps only those points that lie on the selectedcommon plane (relevant common feature points) may be considered. Then,the transformation T_(i,img) that optimizes the placement of the imagefurther may be computed. This can be done either by only considering theselected point-correspondences, or by considering a certain regionaround the point-correspondences, or both.

In the first case a transformation that minimize the distance betweencorresponding points can be computed. In the second case a region aroundthe corresponding points can be defined and the correlation withincorresponding regions can be maximized. Various methods can be used,e.g. simple pixel differencing or optical flow methods. In the lattercase, the transformation based on corresponding points may be used as astarting point which is then further refined by correlation-basedmethods.

From the sensor model, additional information on how accurate themeta-data is were obtained. In this last refinement step, the sensormodel restricts the transformation T_(i,img). This step can beconsidered optional and also depends on the method used to refineT_(i,img). For example if the GPS-position is known to be up to 2 minaccurate point correspondences (relevant common feature points) thatsuggest a position error of 7 m can be ignored. Or for the correlationbased approach the search range can be limited 854 to the accordingrange.

The output of the algorithm described above are two transformationsT_(i,pos) and T_(i,img). The first one can be computed very quicklywhich allows fast placement of the individual images on an overview map(overview image). The latter one includes somewhat more computationaleffort and thus needs more time. However, for interactive systems imagescan be displayed immediately based on the initial transformationT_(i,pos) and later on refined. The final transformation T_(i) is thecombination 860 (re-adjustment of the position of the new image) ofT_(i,pos) and T_(i,img).

In the following, the results of known approaches with are compared withthe proposed hybrid approach. This evaluation mainly focuses on thegeospatial accuracy and image correlation which are specified in aquality metric. Further, the needed computation times of all approachesare compared which have been implemented, for example, in Matlab on astandard PC running at 2.66 GHz.

For the evaluation, a rectangular round trip mission for which 40picture points have been planned is used. Images have been captured froma single UAV flying at an altitude of approximately 30 m. The overlapamong adjacent images is about 60%. A subset of 8 images is used tocompare the stitching results of the known mosaicking approaches.

To evaluate the quality of the different mosaicking approaches thefollowing metric for the overview image quality is defined, for example:

λ(I _(res))=α·λ_(spat)(I _(res))+(1−α)·λ_(corr)(I _(res))  (1)

where

${\lambda_{spat} = {\frac{1}{m}{\sum\limits_{i = 1}^{m}\frac{1}{1 + {\frac{d_{i} - {\hat{d}}_{i}}{d_{i}}}}}}},{\lambda_{corr} = {\frac{1}{n}{\sum\limits_{i = 1}^{n}\frac{1 + {C\; {C\left( {{Overlaps}\left( {I_{{res},{i - 1}},{T_{i}\left( I_{i} \right)}} \right)} \right)}}}{2}}}},{{C\; {C\left( {X,Y} \right)}} = \frac{{Covariance}\left( {X,Y} \right)}{\sigma_{X}\sigma_{Y}}},$

d_(i) is the actual distance measured between two ground control points,{circumflex over (d)}_(i) is the estimated distance extracted fromoverview image and m is the number of considered distances. As it can bededuced from the equations, λ_(spat) and λ_(corr) are all in the rangeof (0, 1]. The total quality function λ is a weighted combination ofλ_(spat) and λ_(corr) (0≦a≦1). λ_(spat) represents the accuracy ofspatial distances while λ_(corr) shows the correlation in areas ofoverlapping images, which is a measure for the seamlessness mosaicking.In this example the weight a=0:5.

Reference Pos Pos + Rot Image Hybrid | P₃P₆ | [m] 31 31.54 30.53 30.1331.30 | P₆P₁₁ | [m] 37.9 38.17 38.07 38.27 38.19 | P₃P₁₁ | [m] 51.7550.61 50.76 50.93 52.40 λ_(spat)(I_(res)) [%] 95.3 96.1 94.6 96.9λ_(corr)(I_(res)) [%] 69.6 74.5 82.4 86.7 λ(I_(res)) [%] 82.4 85.3 88.591.8

The quality of correlation λ_(corr) can easily be noticed in theoverview images in FIG. 2, that is increasing by the complexity of theapproaches.

In the evaluation a triangle is chosen, spanning significant points (P₃;P₆; P₁₁) for simplified spatial evaluation in the reduced set of eightimages (e.g. shown in FIG. 10 a). The table above shows a spatialaccuracy and quality parameters of three known and the proposedmosaicking approaches. In the table the measured distances (| P₃P₆ |, |P₆P₁₁ |, | P₃P₁₁ |), the resulting spatial quality and the correlationquality λ(I_(Res)) are presented and combined according to Equation 1 toa final quality characteristic to compare the presented approaches.Metadata-based approaches, like the position-based approach and theposition-based approach with rotation retain geo-referencing, if onlysimilarity transformations are used. Image-based approaches even whenrestricting the matching function to a similarity transformation show agood correlation quality. The computation time for the whole set of 37images in the scaled resolution of 400 by 300 px took t_(pos)=17:31 sfor position-based, t_(pos+rot)=18:33 s with rotation, and increaseddramatically to t_(image)=459:20 s in the image-based alignmentapproach.

The complete round trip mission is used to evaluate the hybrid approach(FIG. 10 a). However, three images were lost in the real UAV mission(cp. positions B, C and D in FIG. 10 a) which reduced the overlap inthese specific areas to approximately 20%. As shown the deviation of thelast image to the starting point is not noticeable which implies thatthe relative distances are almost kept to a certain extent. Thecomputation time was t_(hybrid)=136:28 s for the whole set of images,which is significantly less than the image based approach. The totalerror range (accuracy information) in the hybrid approach defines thesearch space in order to find the estimated position. By estimating theappropriate image position it is compensated for the total error (GPSand camera tilting errors). FIG. 3 helps to understand this conceptbetter. It is searched inside this possible error range to find the bestestimated position which maximizes the quality function best. The totalerror range used in FIG. 10 a is GPS_(error)+tan(α)×height≅7 m in realworld distance at the ground level, which is approximately equivalent to¼ of the image width. Yet, in a complete nadir view, orthogonality willbe reduced when getting away from the optical axis. Somehow it gives anidea that the middle parts of an images contain more reliable data. Sofor more pleasant result in the mosaicking (generation of the overviewimage), it may be made sure that the central part of each image undereach picture-point is not masked by the border parts of other images.FIG. 11 a shows the relation between correlation of the overlappingparts of two adjacent images in different approaches. As it can be seen,the hybrid approach shows the highest correlation comparing to theothers. FIG. 11 b indicates the relative distance from the estimatedposition to the corresponding GPS position on each image in the hybridapproach. By comparing these two graphs, it can be seen that if theestimated position of an image is close to its indicated GPS position itresults in a higher correlation and vice versa. FIG. 10 b shows asnapshot from a possible user interface. The operator defines the targetarea, then single images are placed iteratively over background map. Theline shows the flight path of the UAV. Any kind of existinggeo-referenced digital maps (e.g., from Google earth or Microsoftvirtual earth) or an empty background can be used.

FIG. 10 a shows an illustration of a mosaicking result of images takenfrom a roundtrip mission using the described concept, while FIG. 10 bshows an illustration of a screen shot of a possible graphical userinterface (GUI) of an unmanned aerial vehicle system. Captured imagesare incrementally stitched over the (partially outdated) backgroundimage.

Some embodiments according to the invention relate to a method forgenerating an overview image from a set of individual images or anincremental mosaicking of images from autonomous, small-scale unmannedaerial vehicles.

Unmanned aerial vehicles (UAVs) have been recently deployed in variouscivilian applications such as environmental monitoring, aerial imagingor surveillance. Small-scale UAVs are of special interest for firstresponders since they can rather easily provide bird's eye view imagesof disaster areas. As described above, a concept to mosaic an overviewimage of the area of interest given a set of individual images capturedby UAVs flying at low altitude is proposed among others. The approachcombines metadata-based and image-based stitching methods in order toovercome the challenges of low-altitude, small-scale UAV deployment suchas non-nadir view, inaccurate sensor data, non-planar ground surfacesand limited computing and communication resources. For the generation ofthe overview image, geo-referencing is preserved as much as possible,since this is an important requirement, for example, for disastermanagement applications. The mosaicking method may be implemented on aUAV system and evaluated based on a quality metric.

Unmanned aerial vehicles (UAVs) are, for example, widely used in themilitary domain. Advances in technology, material science and controlengineering made the development of small-scale UAVs possible andaffordable. Such small-scale UAVs with a total weight of approximately 1kg and a diameter of less than 1 m are getting prominent in civilianapplications too and pose new research questions. These UAVs areequipped with sensors such as accelerometers, gyroscopes, and barometersto stabilize the flight attitude and GPS receivers to obtain accurateposition information. Additionally, UAVs can carry payloads such ascameras, infrared cameras, or other sensors.

Thus, UAVs enable to obtain a bird's eye view of an area which ishelpful in applications such as environmental monitoring, surveillanceand law enforcement, and disaster assessment and disaster management(see for example “M. Quaritsch, E. Stojanovski, C. Bettstetter, G.Friedrich, H. Hellwagner, M. Hofbaur, M. Shah, and B. Rinner.Collaborative Microdrones: Applications and Research Challenges. InProceedings of the Second International Conference on AutonomicComputing and Communication Systems (Autonomics 2008), page 7, Turin,Italy, September 2008”). Obviously, each application domain hasdifferent requirements. One goal is to support first responders indisaster assessment and disaster management since this is the mostchallenging application domain. In disaster situations such asearthquakes or flooding, first responders can not rely on a fixedinfrastructure and the available information (e.g., maps) may no longerbe valid. It is important to provide the first responders a quick andaccurate overview of the affected area, typically spanning hundreds ofthousands of square meters. This overview image may be refined(according to one aspect of the invention) and updated over time and canbe augmented with additional information such as detected objects or thetrajectory of moving objects. When covering large areas at reasonableresolution from such small-scale UAVs, the overview image needs to begenerated from dozens of individual images. Moreover, a number of UAVsequipped with cameras may be employed instead of a single UAV to copewith the stringent time constraints and the limited flight time. TheUAVs, flying at low altitudes of up to 100 m, provide images of theaffected area which are stitched to an accurate overview image.

For example, a hybrid approach which allows to quickly mosaic theindividual images and refine the alignment over time as more images areavailable was described above.

The hybrid approach for image mosaicking may take both the positioninformation and the image data into account. The described mosaickingapproach may be evaluated using a quality metric which is based on aspatial metric and a correlation metric.

For example, a method for generating an overview image (“mosaic”) from aset of individual aerial images is described which have been capturedfrom low altitudes. Position and orientation data from the aerialcameras may be exploited to generate a fast alignment of the individualimages on the ground surface. Further, this alignment may be refined byapplying different alignment procedures such as maximizing the pixelcorrelations within the image overlaps or exploiting estimated depthinformation to identify corresponding points at the same elevationlevel. On the one hand this method is able to provide a quick overviewimage which can be refined later on. On the other hand, it achieves ahigh geo-reference quality and reduces the processing time of therefinement process by limiting the search space through the precedingalignment steps.

In other words, the proposed concept may enable, for example, an onlinemosaicking method that increases the resulting geo-referencing accuracyincrementally. Online execution means that the mosaic is created whileflying and continuously taking images. A first snapshot may be presentedimmediately—potentially with high geo-reference deviation and even fromimages without any overlap. By acquiring more images (with overlap), orcomputing the camera position and view (GPS and IMU) from the image datathe accuracy of mosaicking and geo-referencing increases. In a targetapplication the focus may lie on geo-referencing and projection accuracyinstead of merging or fusing images to a nice looking panorama. Anappealing fusion is optional later. But any waste or reduction of databy fusion may be considered as information about the scene (e.g. movingobjects).

Further, small-scale quadrocopters with worse stability and deviceaccuracy than planes (even small planes) but with big advantages inrespect of the intended use case may be used (e.g. disaster recovery).They are easy to transport and to operate. Additionally, the approachmay have a short feedback loop, determining uncovered or badly coveredareas already during flight, due the mosaicking is done online. The UAVscan react immediately on changes.

In still other words, according to the described concept, a system formosaicking high-resolution overview images of large areas with highgeometric accuracy from a set of images taken from small-scale UAVs maybe realized. This may take the use of small-scale UAVs flying at lowaltitude into account. A hybrid approach is described that combinesinaccurate information on the camera's position and orientation, and theimage data. Thus, it can maintain geometric accuracy and at the sametime enhance the visual appearance. The evaluations show that theapproach results in a higher correlation between overlapping imageregions and retains spatial distances with an error of less than 30 cm.Further, the computation time for a set of 37 images may be reduced byapproximately 70% compared to an image-based mosaicking. More dynamicand interactive methods of mosaicking images may be included to increasethe quality of the overview image, i.e., as new images are taken thetransformation of already mosaicked images are refined. Moreover, theproposed method may be applied also for larger areas and use images frommultiple UAVs.

FIG. 9 shows a block diagram of an unmanned aerial vehicle 900 accordingto an embodiment of the invention. The unmanned aerial vehicle 900comprises a camera 910, a sensor system 920, an accuracy determiner 930and a transmitter 940. The camera 910, the sensor system 920 and theaccuracy determiner 930 are connected to the transmitter 940. The camera910 is able to take an image during a flight of the unmanned aerialvehicle 900. The sensor system 920 determines a position of the unmannedaerial vehicle 900 at the time, when the image is taken, to obtain aposition information associated to the taken image. Further, theaccuracy determiner 930 determines an accuracy of the determination ofthe position of the unmanned aerial vehicle 900 to obtain an accuracyinformation associated to the determined position information. Thetransmitter 940 transmits the taken image together with associatedmeta-data containing the position information of the taken image and theaccuracy information of the position information.

By providing images with position information and accuracy information,an overview image may be generated according to the concept describedabove.

At the same time, the image is taken, means that the determination ofthe position and the accuracy information of the position may betriggered by taking an image. This may be done by a control unit. Theposition and the accuracy information may be determined at the sametime, the image is taken, with a tolerance depending on the systemimplementation (e.g. within 1 s, 100 ms, 10 ms or less).

For example, the sensor system 920 receives a GPS signal (GlobalPositioning System Signal) and determines the position of the unmannedaerial vehicle 900 based on the GPS signal. Further, the accuracydeterminer may determine the accuracy of the determination of theposition also based on the received GPS signal.

The accuracy determiner 930 may store a determined accuracy of thedetermination of the position and provide this accuracy information forall taken images until the accuracy of the determination of the positionchanges more than a threshold.

The transmitter may be a wireless transmitter, so that taken images withassociated meta-data may be transmitted during the flight of theunmanned aerial vehicle 900. Alternatively, the transmitter 940 may be awire bound transmitter comprising a storage device for storing imagesand meta-data during the flight of the unmanned aerial vehicle 900. Inthis example, the taken images and the meta-data may be transmitted(e.g. to an apparatus for generating an overview image of a plurality ofimages) after the unmanned aerial vehicle 900 has landed.

The sensor system 920 may determine additionally a yaw, a nick and aroll of the unmanned aerial vehicle 900 at the time the image is takento obtain a yaw information, a nick information and a roll informationassociated to the taken image. Further, the accuracy determiner 930 maydetermine an accuracy of the determination of the yaw, the nick and theroll of the unmanned aerial vehicle 900 to obtain an accuracyinformation associated to the determined yaw information, the nickinformation and the roll information. The transmitter 940 may transmitthe taken image together with associated meta-data containing theposition information, the yaw information, the nick information and theroll information of the taken image and the accuracy information of theposition information, the yaw information, the nick information and theroll information.

Further, the sensor system 920 may determine one or more environmentalparameters (e.g. wind force). This environmental parameter may betransmitted together with the other meta-data by the transmitter 940 ormay be used by the accuracy determiner 930 to determine the accuracy ofthe determination of the position of the unmanned aerial vehicle 900based on a determined environmental parameter. For example, if there isstrong wind, the accuracy of the determination of the position may belower than under weak wind conditions.

Some embodiments of the invention relate to a method for providing animage together with associated meta-data taken by an unmanned aerialvehicle. The method comprises taking an image during a flight of theunmanned aerial vehicle and determining a position of the unmannedaerial vehicle at a time the image is taken to obtain a positioninformation associated to the taken image as well as determining a yaw,a nick or a roll of the unmanned aerial vehicle at a time, the image istaken, to obtain a yaw information, a nick information or a rollinformation associated to the taken image. Further, the method comprisesdetermining an accuracy of the determination of the position of theunmanned aerial vehicle to obtain an accuracy information associated tothe determined position information and determining an accuracy of thedetermination of the yaw, the nick or the roll of the unmanned aerialvehicle to obtain an accuracy information associated to the determinedyaw information, the nick information or the roll information.Additionally, the method comprises transmitting the taken image togetherwith associated meta-data containing the position information of thetaken image and the accuracy information of the position information,wherein the meta-data further contains the yaw information, the nickinformation, the roll information, the accuracy information of the yawinformation, the accuracy information of the nick information or theaccuracy information of the roll information.

Although some aspects of the described concept have been described inthe context of an apparatus, it is clear that these aspects alsorepresent a description of the corresponding method, where a block ordevice corresponds to a method step or a feature of a method step.Analogously, aspects described in the context of a method step alsorepresent a description of a corresponding block or item or feature of acorresponding apparatus.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a Blue-Ray, a CD, a ROM, a PROM, an EPROM,an EEPROM or a FLASH memory, having electronically readable controlsignals stored thereon, which cooperate (or are capable of cooperating)with a programmable computer system such that the respective method isperformed. Therefore, the digital storage medium may be computerreadable.

Some embodiments according to the invention comprise a data carrierhaving electronically readable control signals, which are capable ofcooperating with a programmable computer system, such that one of themethods described herein is performed.

Generally, embodiments of the present invention can be implemented as acomputer program product with a program code, the program code beingoperative for performing one of the methods when the computer programproduct runs on a computer. The program code may for example be storedon a machine readable carrier.

Other embodiments comprise the computer program for performing one ofthe methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, acomputer program having a program code for performing one of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a datacarrier (or a digital storage medium, or a computer-readable medium)comprising, recorded thereon, the computer program for performing one ofthe methods described herein.

A further embodiment of the inventive method is, therefore, a datastream or a sequence of signals representing the computer program forperforming one of the methods described herein. The data stream or thesequence of signals may for example be configured to be transferred viaa data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example acomputer, or a programmable logic device, configured to or adapted toperform one of the methods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing one of the methods described herein.

In some embodiments, a programmable logic device (for example a fieldprogrammable gate array) may be used to perform some or all of thefunctionalities of the methods described herein. In some embodiments, afield programmable gate array may cooperate with a microprocessor inorder to perform one of the methods described herein. Generally, themethods are advantageously performed by any hardware apparatus.

The above described embodiments are merely illustrative for theprinciples of the present invention. It is understood that modificationsand variations of the arrangements and the details described herein willbe apparent to others skilled in the art. It is the intent, therefore,to be limited only by the scope of the impending patent claims and notby the specific details presented by way of description and explanationof the embodiments herein.

While this invention has been described in terms of several embodiments,there are alterations, permutations, and equivalents which fall withinthe scope of this invention. It should also be noted that there are manyalternative ways of implementing the methods and compositions of thepresent invention. It is therefore intended that the following appendedclaims be interpreted as including all such alterations, permutationsand equivalents as fall within the true spirit and scope of the presentinvention.

1. Apparatus for generating an overview image of a plurality of images,the apparatus comprising: a storage unit configured to store a pluralityof processed images of the overview image, wherein each processed imageof the plurality of processed images is assigned to a position of theoverview image, wherein the storage unit is configured to provide theoverview image comprising the plurality of processed images at theirassigned positions for displaying; and an image processor configured todetermine feature points of a new image and configured to compare thedetermined feature points of the new image with feature points of astored processed image to identify common feature points and to acquire3-dimensional positions of the common feature points, wherein the imageprocessor is configured to determine common feature points locatedwithin a predefined maximum distance of relevance to a reference planebased on the 3-dimensional positions of the common feature points toidentify relevant common feature points, wherein the image processor isconfigured to process the new image by assigning the new image to aposition in the overview image based on a comparison of an imageinformation of each relevant common feature point of the new image withan image information of each corresponding relevant common feature pointof the stored processed image without considering common feature pointslocated beyond the predefined maximum distance of relevance to thereference plane, wherein the storage unit is configured to add theprocessed new image with the assigned position to the plurality ofprocessed images of the overview image.
 2. Apparatus according to claim1, wherein the image processor is configured to determine the referenceplane based on the 3-dimensional positions of the common feature points.3. Apparatus according to claim 2, wherein the image processor isconfigured to determine the reference plane by fitting a horizontalplane to the 3-dimensional positions of the common feature points, sothat a maximal number of common feature points are located within apredefined maximum fitting distance to the reference plane or so thatthe reference plane is as far away as possible from a camera position,the new image was taken from, with at least one common feature pointlocated within the predefined maximum fitting distance to the referenceplane.
 4. Apparatus according to claim 1, wherein the storage unit isconfigured to add the processed new image together with the determinedfeature points of the new image to the plurality of processed images. 5.Apparatus according to claim 1, wherein the image processor isconfigured to determine an overlap region of the new image and thestored processed image, and wherein the image processor is configured tocompare determined feature points of the new image located within theoverlap region with feature points of the stored image located withinthe overlap region, while determined feature points outside the overlapregion are neglected for the identification of the common featurepoints.
 6. Apparatus according to claim 1, wherein the image informationof a relevant common feature point in the new image is a position of therelevant common feature point in the new image and the image informationof a relevant common feature point in the stored image is a position ofthe relevant common feature point in the stored image, wherein the imageprocessor is configured to assign the new image to a position in theoverview image, so that the distance between a position of a relevantcommon feature point in the new image and a position of thecorresponding relevant common feature point in the stored image isminimized or so that a sum of distances between the position of therelevant common feature points in the new image and the position of therespective corresponding relevant common feature points in the storedimage is minimized.
 7. Apparatus according to claim 1, wherein the imageinformation of the relevant common feature point in the new image is anarea of the new image around the relevant common feature point of apredefined size and the image information of a relevant common featurepoint in the stored image is an area of the stored image around therelevant common feature point of the predefined size, wherein the imageprocessor is configured to assign the new image to a position in theoverview image, so that the correlation value of an area around arelevant common feature point of the new image and an area around acorresponding relevant common feature point of the stored image ismaximized or so that a sum of correlation values of areas around therelevant common feature points of the new image and areas around thecorresponding relevant common feature points of the stored image ismaximized.
 8. Apparatus according to claim 6, wherein the imageprocessor is configured to assign the new image to a position in theoverview image based on the minimization of the distance or the sum ofdistances and configured to readjust the assignment of the new image toa position in the overview image based on the maximization of thecorrelation value or the sum of correlation values.
 9. Apparatusaccording to claim 1, comprising an image preprocessor configured topreprocess the new image by assigning the new image to a position in theoverview image based on a position information comprised by meta-data ofthe new image, wherein each image of the plurality of images of theoverview image comprises associated meta-data, wherein the storage unitis configured to store a plurality of preprocessed or processed imagesof the overview image, wherein the image processor comprises an accuracyinformation input for receiving an accuracy information of the positioninformation and a controllable processing engine, wherein the imageprocessor is configured to determine an overlap region of thepreprocessed new image and the stored processed image within theoverview image based on the assigned position of the preprocessed newimage and the assigned position of the stored processed image, whereinthe controllable processing engine is configured to process thepreprocessed new image by readjusting the assigned position of thepreprocessed new image based on the comparison of the image informationof each relevant common feature point of the preprocessed new image withthe image information of each corresponding relevant common featurepoint of the stored processed image, wherein the controllable processingengine is controlled by an accuracy information of the positioninformation received by the accuracy information input, so that amaximal readjustment of the assigned position of the preprocessed newimage is limited based on the received accuracy information of theposition information.
 10. Apparatus according to claim 9, wherein theimage preprocessor is configured to preprocess the new image bycorrecting an orientation and a perspective distortion of the new imagebased on a roll information, a pitch information and a yaw information,wherein the roll information, the pitch information and the yawinformation is comprised by the meta-data of the new image. 11.Apparatus according to claim 9, wherein the storage unit is configuredto add the preprocessed new image with the assigned position to theplurality of preprocessed or processed images, and wherein the storageunit is configured to provide the overview image comprising thepreprocessed new image at the assigned position for displaying beforethe preprocessed new image is processed by the image processor. 12.Apparatus according to claim 9, wherein the image preprocessor isconfigured to receive successively a plurality of new images andconfigured to preprocess each received new image, wherein the storageunit is configured to store each preprocessed new image and configuredto provide an updated overview image after storing a predefined numberof preprocessed new images.
 13. Apparatus according to claim 9,comprising an unmanned aerial vehicle control unit configured togenerate a control signal for piloting an unmanned aerial vehicle to aregion corresponding to an area of the overview image being not coveredby an image of the plurality of preprocessed or processed images. 14.Method for generating an overview image of a plurality of images,comprising: storing a plurality of processed images of the overviewimage, wherein each processed image of the plurality of processed imagesis assigned to a position in the overview image; determining featurepoints of a new image; comparing the determined feature points of thenew image with feature points of a stored processed image to identifycommon feature points and to acquire 3-dimensional positions of thecommon feature points; determining common feature points located withina predefined maximum distance of relevance to a reference plane based onthe 3-dimensional positions of the common feature points to identifyrelevant common feature points; processing a new image by assigning thenew image to a position in the overview image based on a comparison ofan image information of each relevant common feature point of the newimage with an image information of each corresponding relevant commonfeature point of the stored processed image without considering commonfeature points located beyond the predefined maximum distance ofrelevance to the reference plane; adding the new image with the assignedposition to the plurality of processed images of the overview image;providing the overview image comprising the plurality of processedimages at their assigned positions for displaying.
 15. A non-transitorycomputer readable medium including a computer program with a programcode for performing a method, when the computer program runs on acomputer or a micro controller, for generating an overview image of aplurality of images, comprising: storing a plurality of processed imagesof the overview image, wherein each processed image of the plurality ofprocessed images is assigned to a position in the overview image;determining feature points of a new image; comparing the determinedfeature points of the new image with feature points of a storedprocessed image to identify common feature points and to acquire3-dimensional positions of the common feature points; determining commonfeature points located within a predefined maximum distance of relevanceto a reference plane based on the 3-dimensional positions of the commonfeature points to identify relevant common feature points; processing anew image by assigning the new image to a position in the overview imagebased on a comparison of an image information of each relevant commonfeature point of the new image with an image information of eachcorresponding relevant common feature point of the stored processedimage without considering common feature points located beyond thepredefined maximum distance of relevance to the reference plane; addingthe new image with the assigned position to the plurality of processedimages of the overview image; providing the overview image comprisingthe plurality of processed images at their assigned positions fordisplaying.